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ABSTRACT 



In educational effectiveness research, multilevel models are 
increasingly used because these models take the multilevel structure of the 
data into account. In this paper, the effect of ignoring one or more levels 
of variation in hierarchical linear regression is explored, using a model 
with four hierarchical levels (individual pupil, class group, teacher, and 
school) as a reference model. A distinction was made between ignoring the top 
or intermediate levels on the fixed and random parameters of different random 
intercept models with a real data set consisting of 2,680 Flemish (Belgian) 
secondary school students. Results show that ignoring the top level causes an 
overestimation of the variance belonging to the highest level considered. 
Ignoring an intermediate level causes an overestimate of the variance 
belonging to the level just above and that just below the level considered. 
The standard error of the variance estimate of the highest level considered 
(ignoring a top level) or of the level just under the ignored intermediate 
level is overestimated, while the standard error of the variance estimate to 
the intercept estimate seems to be underestimated in models with ignored 
levels. In addition, ignoring the top level can cause unstable regression 
coefficient estimates for the exploratory variables belonging to the highest 
level considered, while ignoring an intermediate level can cause unstable 
regression coefficient estimated of the exploratory variables belonging to 
the level just above and just under the level ignored. It is concluded that 
ignoring an important top or intermediate level can lead to different 
research conclusions. An appendix contains a list of the variables 
considered. (Contains 13 tables and 9 references.) (SLD) 
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ABSTRACT ^ 

^ ( 

In educational effectiveness research, multilevel models are increasingly used because these 
models take the multilevel structure of the data into account. In this paper, the effect of 
ignoring one or more levels of variation in hierarchical linear regression analysis is 
explored. We used a model with four hierarchical levels (the individual pupil, the class group, 
the teacher and the school) as reference model and we made a distinction between ignoring 
the top and the intermediate levels. We explored the effects of ignoring the top or the 
intermediate levels on the fixed and random parameters of different random intercept models 
by means of a real data set. The results show that ignoring a top level causes an 
overestimatiqn of the variance belonging to the highest level considered. The variance of the 
other levels is unaffected. Ignoring an intermediate level causes an overestimation of the 
variance belonging to the level just above and the level just under the level ignored. Thus, 
ignoring levels results in a different attribution of the variance to the levels. The standard 
error of the variance estimate of the highest level considered (ignoring a top level) or of the 
level just under the ignored intermediate level is overestimated, whereas the standard error of 
variance estimate of the level just above the ignored intermediate level is underestimated. 

Also the standard error of the intercept estimate seems to be underestimated in models with 
ignored levels. It is notable that the regression coefficient estimates of the explanatory 
variables 'at the affected level(s) ' in models ignoring one or two levels differ from those 
obtained from the four-level model. However, the estimates of exploratory variables of the 
non-affected levels do not differ from those of the four-level model considered. Ignoring the 
top level can cause unstable regression coefficient estimates for the exploratory variables 
belonging to the highest level considered, while ignoring an intermediate level can cause 
unstable regression coefficient estimates of the exploratory variables belonging to the level 
just above and the level just under the level ignored. We conclude that ignoring an important 
top or intermediate level can lead to different research conclusions. 



INTRODUCTION 

In educational effectiveness research, multilevel models are increasingly used because these 
models take the multilevel structure of the data irito account. In multilevel research, the data 
structure in the population is often hierarchical (e.g., pupils in schools) and the sample data are 
viewed as a multistage sample from their hierarchical population. In such samples, the 
individual observations are generally not completely independent because of selection 
processes and because of the common history they share by belonging to the same group (Hox, 
1994). As a result, the traditional OLS regression analysis (and ordinary significant tests) with 
the individual as the unit of analysis may not be used because the important assumption of 
independence of residual error terms is violated (Hox & Kreft, 1994). Multilevel models take 
these dependencies into account 

The identification of a level above the individual level implies the existence of different levels 
of variation: if there are effects of the social context (the group level) on individuals, these 
effects must be mediated by intervening processes that depend on the characteristics of the 
social context. Groups can have'~a main effect on individuals (different intercepts) or there can 
be cross-level interaction effects (different slopes). The latter requires the specification of 
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processes within individuals that cause those individuals to be differentially influenced by 
certain aspects of the context. 

However, what are the criteria for choosing levels and the number of levels? There are three 
kinds of criteria; the theory under investigation (Hox, 1994; Snijders & Bosker, 1998) or the 
research question; the kind of sampling used (cf. multistage sampling) (Hox, 1994; Snijders & 
Bosker, 1998); or the number of units belonging to a level e.g., when there are only three units 
in the highest level, it makes no sense to consider that level in the analysis. Sometimes, none of 
the criteria can be used to determine the (number of) levels. The international literature about 
multilevel modeling has paid a lot of attention to the importance of identifying a level above 
the individual level e.g., the comparisons between multilevel modeling and ordinary least 
squares regression analysis (Goldstein, 1995; Longford, 1993). However, very few or no 
attention is paid to the determination of (the number of) levels or to the effect of ignoring one 
or more levels in the analysis. 

PURPOSE OF THE STUDY 

In this paper, the effect of ignoring one or more levels of variation in hierarchical linear 
regression analysis is explored. We used a model with four hierarchical levels (the individual 
pupil, the class group, the teacher and the school) as the reference model (PCTS). We made a 
distinction between ignoring the highest level (the school level), the highest two levels (the 
school and the teacher level) and ignoring one or two intermediate levels (the teacher and/or 
class level). This resulted in six other models: the pupil-class-teacher model (PCT), the pupil- 
class-school model (PCS), the pupil-teacher-school model (PTS), the pupil-class model (PC), 
the pupil-teacher (PT) and the pupil-school model (PS). The effects of ignoring levels on the 
fixed and random parameters of multilevel models are addressed. In this contribution we 
restrict the e;|cploration to random intercept models. 



DATA SOURCES AND METHOD USED 

The reported data in this contribution stem from the ‘Longitudinaal Onderzoek Secundair 
Onderwijs (LOSO)’ project (longitudinal research in secondary education project) of Van 
Damme et al. (1997a,b) and is funded by the Department of Education of the ministry of the 
Flemish Community. The data set consists of longitudinal data on pupils and secondary schools 
in Flanders (Belgium) studied for a period of seven years. The sample of pupils is taken from 
almost all schools with a first grade in three regions. The schools are to a certain extent 
representative of the schools in Flanders: the courses offered and the distribution of the pupils 
over the courses of study in these three regions together are comparable to the situation in 
Flanders. We used information on the individual pupil (numerical intelligence), the class group 
(mean numerical intelligence), the teacher level (mathematics teacher's reports of instructional 
approach) and the school level (teachers’ reports of instructional approach and school life). An 
overview of the variables is given in the appendix. In our analyses, we used a sample of 2680 
pupils following the general track in the first year of secondary education (grade 1 A) and 
belonging to 150 classes, 81 mathematics teachers and to 46 secondary schools. All pupils 
entered secondary education for the first time. Mathematics achievement at the end of the 
common first grade (general track) is the dependent variable in our analyses. 
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To analyze the data, the computer program MLn (Rasbash & Woodhouse, 1995) for multilevel 
analysis was used. By manipulating the number of levels and the kind of level ignored (top 
level or intermediate level) and comparing the results of these models with the four-leyel 
model, we investigated the implications of ignoring levels of variation. Firstly, the implications 
of ignoring levels on the variance structure will be addressed. The null-model solution of a 
four-level model will therefore be compared with the solutions of different three- and two-level 
models. Secondly, explanatory variables at different levels will be added to the models. The 
three- and two-level models will be analyzed and compared with the four-level model to show 
the effect of ignoring levels on the fixed regression coefficients of independent variables. To 
compare the fixed and random parameters of the three- and two-level models with the 
parameters of the four-level models, we calculated the difference between the relevant 
parameter of the four-level model and the parameter of the model we wanted to compare this 
with. We divided this difference by the standard error of the parameter of the four-level model. 
^This procedure is comparable with calculating confidence intervals around the parameter 
estimate of the four-level model. It gives an impression of the difference between numerical 
values from a statistical point of view. 



RESULTS 

The analysis of the four-level null model revealed that each level is important: 54.42% of the 
overall variance in mathematics is linked to the individual level, 14.71% to the class level, 
18.07% to the teacher level and 12.81% to the school level. The four-level null model fits the 
data better than all the other null models do; the difference between deviance of the four-level 
model and of the other models is always significant at least at the five-percent level (see tables 
2 and 3). 

(Insert table 1) 



Effects of Ignoring Levels on the Fixed and Random Parameter Estimates of Null Models 

In a first step we investigated the effects of ignoring one or two top levels . We used three 
models: the pupil-class-teacher-school model (PCTS), the pupil-class-teacher model (PCT) and 
the pupil-class model (PC). First, null models (with no explanatory variables) are fitted to, 
provide estimates of the components of the total variation at each level. 

(Insert table 2) 

Table 2 shows that ignoring a top level causes an overestimation of the variance belonging to 
the highest level considered. The variance of the other levels is unaffected. The standard error 
of the intercept seems somewhat underestimated in models where the top level is ignored. This 
is true in particular for the PC model. In contrast with this, the standard error of the variance 
estimate belonging to the highest level considered is overestimated. Again, the greatest 
differences are found between the four-level model and the PC model. 



' A similar method was used by Suijders and Bosker (1998). They compared the parameter estimates of a 
random intercept model with the results of an ordinary least squares (OLS) regression analysis. 



Ignoring an intermediate level (see table 3) causes an overestimation of the variance belonging 
to the level just above and the level just under the level ignored. As with the results of ignoring 
a top level, the standard error of the intercept seems to be somewhat underestimated in the 
models where an intermediate level is ignored. The standard error of the variance estimate 
belonging to the level just under the intermediate level ignored is somewhat overestimated, 
while the standard error of the variance estimate belonging to the level just above the 
intermediate level ignored is somewhat underestimated. The greatest differences were found 
between the four-level model and the models where the class or the class and the teacher level 
is ignored (the PTS and the PS model). 

(Insert table 3) 

Thus, ignoring levels results in a different attribution of the variance to the levels considered 
and can therefore lead to different research conclusions. 



Effects of Ignoring Levels on the Parameter Estimates of Models with Explanatory Variables 

In a second step of our exploration, explanatory variables were added to the models. All 
independent explanatory variables were centered around the grand mean of their corresponding 
level. By doing so, the computer calculations are easier and the parameters are estimated at 
sensible choices of locations without modeling a different model than the raw scores model 
(OpdenakJcer & Van Damme, 1997). 

In the first part of this paragraph, we vrill discuss the effects of ignoring one or two top levels 
on the fixed parameter estimates of exploratory variables. Secondly, the standard errors of the 
fixed parameter estimates are investigated. In the last part; the random parameter estimates are 
addressed. The same approach is used for the discussion of the effects of ignoring one or two 
intermediate levels. 



The case of ignoring top levels 

First, the numerical intelligence of the pupils (NIQ) was added at the pupil level to the PCTS 
model and the PCT and the PC model (see table 4). Next, the class mean numerical intelligence 
(CLNIQ) at the class level was also added to the models (see table 5). The PCTS and PCT 
models were also fitted with NIQ, CLNIQ and teacher variables together (see table 6), and 
with NIQ and teacher variables alone (see table 7). 

Tables 4-7 indicate that the parameter estimate of the pupil-level variable NIQ was never 
affected by ignoring one or two top levels. 

(Insert table 4) 

The parameter estimate of the class-level variable CLNIQ was only once weakly 'affected' (see 
table 5 and 6). This was in the case of the PC-model with ignoring the school and the teacher 
level. Remember that we found also that the variance of the class level differed between the 
four-level null model and the PC null model. 

(Insert table 5) 
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Tables 6 and 7 indicate that the numerical values of the parameter estimates of the teacher- 
level variables can be different between the four-level. model and the model ignoring the school 
level (see tables 6 and 7). From a statistical point of view we see that the estimates of a few 
teacher variables are weakly affected.^ Also in this case the variance at the teacher level 
differed between the four-level null model and the null model ignoring the school level. 

(Insert table 6 and 7) 

The standard errors of the explanatory variable estimates are hardly affected by ignoring a top 
level. In accordance with the conclusions of the null models, the standard errors of the 
intercept estimates seem somewhat underestimated. The estimates of the variance belonging to 
the highest level considered are overestimated and the corresponding standard errors are also 
somewhat overestimated. 



The case of ignoring, intermediate levels 

In tables 8 until 13 the effects of ignoring intermediate levels on the fixed parameter estimates 
of exploratory variables are presented. 

In three models i.e. PS-, PTS- and PT-model, the estimate of the NIQ pupil variable regression 
coefficient is strongly different from the estimate obtained in the four-level model (see table 8, 
11,12 and 13). In all cases the estimates of the models ignoring levels differ by more than two 
standard errors from the estimate of the four-level model. Table 3 showed that the variance at 
the pupil level of the corresponding null models differed from the four-level null model. 

(Insert table 8) 

The class-level variable CLNIQ had a slightly different regression coefficient in the four-level 
model than in the model ignoring the teacher level (see tables 9 and 10). 

(Insert table 9 and 10) 

The majority of the teacher-level variables estimates of the models ignoring the class and/or the 
school level differed numerically from the ones obtained with the four-level model, (see tables 
1 1 and 12). Sometimes, a variable is significant in the four-level model e.g., 'TSUBJMOR' is 
significant at 0.03 level one sided test, while this variable is not significant at all (p=0. 117 one 
sided test) in the PT-model (see tablel2). However, from a statistical point of view, only a few 
school-level variables have different estimates. Because of the limited number of teachers in 
our sample, we think that the differences between the models may be underestimated. 

(Insert tables 1 1 and 12) 

Information about the school-level variables estimates are given in tables 12 and 13. In this 
case it is possible to compare the estimates of the four-level model with models with 'affected' 
•school-level variance (models PCS and PS) on the one hand and with a model without affected 

^ It is possible that the estimates of the teacher-level variables seem only weakly affected because of 'large' 
standard errors. These large standard errors could be caused by the limited number of teachers (n=81) in our 
sample because the standard error, in addition to the standard deviation, is also function of the sample size. 
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school-level variance (PTS) on the other. The differences in school-level variables estimates 
are the greatest between models with 'affected' school-level variance and the four-level model. 
Again, from a statistical point of view, only a few estimates differ between the models. Our 
remark in the case of teacher variables is even more true here: the limited number of schools 
(n=46) could possibly cause an underestimation of the differences between the estimates. 

As with ignoring a top level, ignoring an intermediate level does not affect the standard errors 
of the explanatory variable estimates. Only the standard error .of the intercept is somewhat 
underestimated. The estimates of the variance belonging to the level just above and just under 
an intermediate level ignored are overestimated. The standard error of the variance estimate 
just under the level i^ored is also overestimated, whereas the standard error of the variance 
estimate just above the level ignored is underestimated. 



CONCLUSIONS 

In this study we explored the effects of ignoring top or intermediate levels on the fixed and 
random parameters of different random intercept models. The results show that ignoring the 
top level causes an pverestimation of the variance belonging to the highest level considered. 
The variance of the other levels is unaffected. Ignoring an intermediate level causes an 
overestimation of the variance belonging to the level just above and the level just under the 
level ignored. The standard error of the variance estimate of the highest level considered 
(ignoring a top level) or of the level just under an ignored intermediate level is overestimated, 
whereas the standard error of the variance estimate of the level just above the level ignored is 
underestimated. Also the standard error of the intercept estimate seems underestimated in 
models with ignored levels. Thus, ignoring levels results in a different attribution of the 
variance to the levels. A^noteworthy result is also the finding that the regression coefficient 
estimates of the explanatory variables 'of the affected level(s)' in models ignoring one or two 
levels differ fi-om the ones obtained from the four-level model. However, the estimates of 
exploratory variables of non-affected levels do not differ from the ones of the four-level model 
considered. Ignoring a top level can cause unstable regression coefficient estimates of the 
exploratory variables belonging to the highest level considered, while ignoring an intermediate 
level can cause unstable regression coefficient estimates of the exploratory variables belonging 
to the level just above and the level just under the ignored level. Based on our research, 
ignoring an important top or intermediate level could lead to different research conclusions. 
Therefore, it is recommended that users of multilevel analysis pay enough attention to the 
determination of (the number) levels. • 

Our study was a first attempt to investigate the effects of ignoring levels. There are a few 
limitations e.g. , we did not use simulation research, our sample of teachers and schools was 
limited and we restricted our investigation to random intercept models. However, we used a 
real data set and could therefore see the possible implications of ignoring a level in a real 
school effectiveness research study. The results make clear the importance of further research 
on this topic. We make a plea for simulation research (with larger data sets on the levels above 
the pupil level) to investigate the circumstances under which ignoring levels can cause wide 
differences between a model with all the levels taken into account and models ignoring one or 
more levels. 
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Table 1 . Results for the four-level null model 




PCTS 

Estimate 


SE 


% total variance 


Fixed parameters 
Intercept 


43,08 


1.235 




Random parameters 
School level 


29,11 


16,42 


12,81 


Teacher level 


41,08 


15.65 


. 18,07 


Class level 


33,43 


6,879 


14,71 


Pupil level 


123,7 


3,478 


54,42 


Deviance 


20884.9 







Table 2. Results for the four-level null model and for models ignoring one or two top levels 





PCTS 

Estimate 


SE 


PCT 

Estimate 


SE 


PC 

Estimate 


SE 


Fixed parameters 














Intercept 


43,08 


1.235 


43,65 


1.095 


44.12 


0.8648VV 


Random parameters 














School level 


29,11 


16,42 










Teacher level 


41.08 


15.65 


71.54***(*) 


15,69 






Class level 


33,43 


6.879 


33.43 


6,89 


104.7**** 


12.94AA 


Pupil level 


123.7 ■ 


3,478 


123.7 


3.478 


123,7 


3.478 


Deviance 


20884,9 




20890,1 




20929.1 





Diff= I estimatepcTs - estimateoMi /SEpcts 

. 0,675 < Diff< 0.84 (0.5 > p > 0.4 two sided test); ° 0.84 < Diff< 1,035 (0.4 > p > 0.3); ♦ 1.035 < Diff< 1.28 
(0.3>p>0.2); ♦♦ 1.28<Diff< 1.645 (0,2 > p> 0.1); ♦♦♦ 1.645<Diff< 1.96 (0.1 > p >0.05); 

♦*♦(♦) Diff = 1 .946 (p = 0.052); ♦♦♦♦ Diff> 1 .96,(p < 0,05) 

V 0.75 < SEpcTs/SEoM < 0.80; W SEpcts/SEom £ 0.75; A 1,25 > SEpcts/SEom ^ 1,20; AA SEpcts/SEom >1-25 
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Table 5. Results for the four-level model and for models ignoring one or two top levels (pupil-level 



variable and class-level variable added) 





. PCTS 
Estimate 


SE 


PCT 

Estimate 


SE 


PC 

Estimate 


SE 


Fixed parameters 














Intercept 


45.22 


0.7029 


45.22 


0.6255 


45.36 


0.4888VV 


NIQ 


0.4285 


0.01804 


0.4284 . 


0.01804 


0.4286 


0.01805 


CLNIQ 


0.5242 


0.05887 


0.5232 


0.05788 


0.5661. 


0.05709 


Random parameters 














School level 


7.644 


5.351 










Teacher level . 


14.89 


5.624 


22.04*(*) 


5.114 






Class level 


9.032 


2.509 


9.078 


2.521 ^ 


29.57**** 


4.112AA 


Pupil level 


101.1 


2.841 


101.1 


2.841 


101.1 


2.841 


Deviance 


20218 




20219.5 




20245.5 




Diff = 1 estimatepcTS - estimateoMi /SEpcts 












. 0,675 < Diff< 0.84 (0.5 > p > 0.4 two sided test); ° 0.84 < Diff< 1.035 (0.4 > p > 0.3); * 1.035 < Diff< 1.28 (0.3 > 


P> 


0.2); *(*)Diff = 1.27 (p 


= 0.102); ** 1.28'< Diff< 1.645 (0.2 > 


p>0.1);*** 1.645 <Diff< 1.96 (0.1 


> p>0.05; ****Diff 


> 1.96 (p<0.05) 














V 0.75 < SEpcTs/SEoM ^ 0.80; VV SEpcts/SEom < 0.75; A 1.25 > SEpcts/SEom > 1.20; AA SEpcts/SEom > 1.25 





Table 6. Results for the four-level model and for models ignoring one or two top levels (pupil-level 
variable, class-level variable and teacher variables added) 




PCTS 

Estimate 


SE 


PCT 

Estimate 


SE 


Fixed parameters 








' 


Intercept 


45.1 ' 


0.7171 


45.2 


0.5843VV 


NIQ 


0.4285 


0.01804 


0.4285 


0.01804 


CLNIQ 


0.5422 


0.0601 


0.5359 


0.0588 


TJOBSAT 


1.184 


1.372 


1.339 


1.444 


TPUPILACT 


0.05985 


1.328 


-0.9379. 


1.349 


TCLASSMAN 


0.5931 


1.484 


0.2753 


1.545 


TPERSRELP 


-0.7458 


1.689 


-1.052 


1.831 


TDIFF 


1.676 


1.433 


1.936 


1.492 


TCONTR 


-2.117 


1.159 


-imi 


1.293 


TPERSDEV 


2.202 


1.771 


2.125 


2.026 


TSUBJMOR 


4.926 


1.454 


4.155 


1.626 


TDISCIPL 


-0.3549 


1.468 


0.1014 


1.622 


TPOSDEFF 


-0.06666 


1.463 


-0.06714 


1.578 


Random parameters 










School level 


13.1 


' 5.124 






Teacher level 


6.221 


3.732 , 


17 93**** 


4.471 


Class level 


9.085 


2.499 


9.063 


2.511 


Pupil level 


101.1 


2.841 


101.1 


2.841 


Deviance 


20202.3 




20207.8 




Diff = 1 estimatepcTs - estimateoMi /SEpcts 








. 0,675 < Diff< 0.84 (0.5 > p > 0.4 two sided test); ° 0.84 < Diff< 1.035 (0.4 > p > 


0.3);* 1.035 < Diff < 1.28 (0.3 >p> 


0.2); ** 1.28 < Diff< 1.645 (0.2 > p > 0.1); *** 


L645 < Diff< 1.96 (0.1 > p >0.05; 


****Diff> 1.96 (p<0.05) 


V 0.75 < SEpcts/SEom < 0.80; VV SEpcts/SEom < 0.75; A 1.25 > SEpcts/SEom > ,1.20; AA SEpcts/SEom > 1.25 
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Table 7. Results for the four-level model and for models ignoring one or two top levels (pupil-level 



variable and teacher, variables added) 





PCTS 

Estimate 


SE 


PCT 

Estimate 


SE 


Fixed parameters 










Intercept 


44.29 


0.8903 


44.77 


0.7491 


NIQ 


0.4606 


0.01753 


0.4614 


0.01751 


TJOBSAT 


1.881 


1.756 


2.244 


1.85 


TPUPILACT 


0.7435 


1.693 


-1.146* 


1.732 


TCLASSMAN 


1.042 


1.905 


0.6214 


1.987 


TPERSRELP 


-0.2488 


2.189 


-0.7093 


2.364. 


TDIFF 


1.349 


1.843 


1.574 


1.918 


TCONTR 


-4.324 


1.472 


-4.199 


1.636 


TPERSDEV 


0.4627 


2.289 


1.37 


2.602 


TSUBJMOR 


3.522 


1.878 


1.908° 


2.067 


TDISCEPL 


-0.1758 


1.899 


0.2854 


2.083 


TPOSDIFF 


-0.7508 


1.887 


-1.631 


2.022 


Random parameters 










School level 


19.2 


8.139 






Teacher level 


13.42 


6.377 


3 1 78**** 


7.335 


Class level 


15:33 


3.573 


15.34 


3.586 


Pupil level 


101.1 


2.843 


101.2 


2!844 


Deviance 


20268 




20274.6 





Diff= I estimatepcTs - estimateoMi /SEpcrs 

. 0,675 < Diff< 0.84 (0.5 > p > 0.4 two sided test); “ 0.84 < Diff< 1.035 (0.4 > p > 0.3); • 1.035 < Diff< 1.28 (0.3 > p 
0.2); •• 1.28<Diff< 1.645 (0.2>p>0.1); ••• 1.645 <Diff< 1.96 (0.1 > p >0.05; •••• Diff> 1.96 (p< 0.05) 

V 0.75 < SEpcts/SEom 5 0.80; VV SEpcts/SEom S 0.75; A 1 .25 > SEpcts/SEom > 1 .20; AA SEpcts/SEom > 1 .25 
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Table 8, Results for the four-level model and for models ignoring one or two intermediate levels (pupil-level variable added) 

PCTS PCS PS . . PTS PT 

Estimate 5E Estimate SE Estimate SE Estimate 5E Estimate SE 

Fixed parameters 

Intercept 44.55 0.8885 44.38 0.8854 44.68 0.8575 44.71 0.8678 44.9 0.7812 
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Table 10. Results for the four-level model and for models ignoring one or two intermediate levels (pupil-level variable, class-level variable and school-1 

variables added) ^ 

PCTS PCS 

Estimate ' SE Estimate , SE 
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Table 11. Results for the four-level model and for models ignoring one or two intermediate levels (pupil-level variable and teacher-level variables 
PCTS PTS PT ■ 

■ Estimate SE Estimate SE Estimate 

Fixed parameters 

Intercept 44.29 0.8903 44.47 0.8663 44.86 0.7307V 
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Table 12. Results for the four-level model and for models ignoring one or two intermediate levels (pupil-level variable, teacher-level variables and school-1 

variables added) . ' 

' PCTS PTS 

Estimate SE ^ Estimate SE 

Fixed parameters 
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Table 13, Results for the four-level model and for models ignoring one or two intermediate levels (pupil-level variable and school-level variables added) 

PCTS PCS 'PS PTS 

^ Estimate Estimate Estimate S£ Estimate SE ^ 

Fixed parameters 

Intercept 44.53 0.7886 44.43 0.792 44.8 0.771 44.68 0.7756 
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APPENDIX LIST OF VARIABLES 

Dependent Pupil Variable 
PERCWEl 

Percentage of points on a mathematics achievement test measured at the end of the first grade of secondary 
education. The test is curriculum based and approved by a board of inspectors and teachers. 

The reliability is 0.80. 

Explanatory Variables 
* Pupil-level variable 
NIQ 

Numerical intelligence measured at the beginning of the school year. The test contains scales of different 
Dutch and Flemish instruments. The reliabilities of the subscales range from 0.82 to 0.92. 



* Class-level variable 
CLNIQ 

Mean of the numerical intelligence of all the pupils in Ae class group. 



* Teacher-level variables 



Name 


Description 


Number of items 
and Reliability 


T JOBS AT 


Job satisfaction of the teacher 


n=14 

a= 0.87' V 


TPUPILACT 


Amount of active pupil participation in the course 


n=7 






a=0.77 


TCLASSMAN 


Amount of orderly classroom management 


n=6 

a= 0.74 


TSUBJMOR 


Orientation on subject matter acquisition 


n=9 

a= 0.70 


TDISCIPL 


Focus on discipline and obedience 


n=9 

a= 0.70 


TCONTR 


Use of control activities (to have the pupils under 


n=3 - 




control) > 


a= 0.50 


TDIFF 


Amount of differentiation activities and material 


n=6 

a= 0.67 


TPERSRELP 


Personal relationship with pupils based on trust 


n=6 




■ 


a= 0.67 


TPERSDEV 


Orientation on the development of the person(ality) of 


n=14 




■ the pupils 


a= 0.77 


TPOSDIFF 


Positive attitude towards differentiation 


n=7 

a= 0.68 
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^ School-level variables 



The school-level variables were constructed on the basis of the reports of a representative sample of 
teachers on a school characteristics questionnaire. First, orthogonal factor analyses were run to construct 
scales on the teacher level. We used the mean for the teacher sample of each school (on these scales) to , 
construct school variables. To reduce the number of variables involved, we ran a second order principal 
component analysis with varimax rotation. 

Results of the Component Analysis of Second Order 



SCOOPTC: Co-operation of teacher staff in relation to teaching methods and pupil counseling 



Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


AUTTMETH 


-0.82 


Autonomous versus collective 
decision making about teaching 
methods 


n=8 

a= 0.67 


CSTMETH 


0.79 


Amoimt of consult with subject 
teachers about teaching methods 


n=10 

a=0.86 


PCOUNSEL 


0.66 


Amount of pupil counseling activities 


n=7 

a= 0.72 


FORMSTRR 


0.66 


Amount of formal structure and rule 
setting 


n=ll 
a= 0.69 


PUPILRET 


-0.54 


Pupil retention orientation 


n=2 

a= 0.70 


POSPMS 


0.52 


Positive attitude towards extern pupil 
coimseling center 


n=8 

a= 0.89 


FUNCTIORG 


0.41 


Functioning of the school as an 


n=34 






organization 


a=0.94 



SDISCSMACQ: Focus on discipline and subject matter 




Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


SUBJMOR 


0.87 


Orientation on subject matter 
acquisition 


n=9 

a= 0.70 


SOCEDUC 


-0.70 


Importance of social education 


n=7^ 


DISCIPL 


0.55 


Focus on discipline and obedience 


n=9 

a= 0.70 


CONTR 


0.53 


Use of control activities (to have the 
pupils under control) 


^ n=3 
a= 0.50 



' Eight curriculum goals (social education, intellectual education, creativity development, cultural education, 
emotional development physical education, moral education, vocational training) were presented in pairs. All 
possible combinations were presented. The respondents had to choose for one goal of each pair. Also an indication 
of the importance of their choice (much more important, more important, a little bit more important) had to be 
made 
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SDIFF: Attention to pupil differences 



Scales of factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


ACTPROBL 


0.74 


Activities undertaken when 
confronted with a problem pupil 


n=5 

a= 0.64 


DIFF 


0.70 


Amount of differentiation activities 
and material 


n=6 

a= 0.67 


PUPILACT 


0.64 


Amount of active pupil participation 
in the course 


n=7 

a= 0.77 


CFTPUPIL 


0.63 


Amount of consult with form teachers 
about pupil affairs 


n=14 
a= 0.84 


SPERSP: Focus on personality development and personal relationships with pupils 


Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


PERSRELP 


0.78 


Personal relationship with pupils 
based on trust 


n=6 

a= 0:67 


PERSDEV 


0.63 


Orientation on the development of the 
person(ality) of the pupils 


n=14 
a= 0.77 


POSPARENT 


0.64 


Positive attitude towards parents and 
parental involvement in the school 


n=10 

a= 0.68 ’ 


USETRES 


0.43 


Use of test results 


n=l 



SOLENVICE: Orderly learning environment focused on intellectual and cultural education 



Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


CLASSMAN 


0.72 


Amount of orderly classroom 
management 


n=6 

a= 0.74 


CULTEDUC 


0.70 


Importance of cultural education 


n=7 


JOBSAT 


0.54 


Job satisfaction 


n=14 
a= 0.87 


INTELEDUC 


0.53 


Importance of intellectual education 


n=7 


HIETRACK 


-0.38 


Amount of hierarchy between the 
tracks in the school 


n=5 

a=0.71 


SCREAIND: Focus on creativity and individual 




Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


CREA 


0.81 


Importance of developing creativity 


n=7 


POSDIFF 


0.62 


Positive attitude towards 
differentiation 


n=7 

a= 0.68 


TIMESCH 


0.38 


Amount of time spend on school work 


n=l 
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SEMDMEDU: Focus on education (emotional development and moral education) versus vocational 
training 



Scales of Factor Analysis 


Component 

loadings 


Description 


Number of items 
and Reliability 


EMODEV 


0.78 


Importance of emotional development 


n=7 


FYSEDUC 


-0.66 


Importance of physical education 


n=7 


MOREDUC 


0.60 


Importance of moral education 


n=7 


VOCTRAIN 


-0.51 


Importance of vocational training 


n=7 
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